{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Homework.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Afanasyy/colab/blob/main/Homework.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Задание**: на месте `# ваш код` самостоятельно впишите код, соответствующий заданию.\n",
        "Все аутпуты (выходные данные) предаставлены для подсказки. Ваши выходные данные могут отличаться по виду, но не по содержанию."
      ],
      "metadata": {
        "id": "6Zt9MXHgyzXT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 1. spaCy"
      ],
      "metadata": {
        "id": "TXijXcb8OQA1"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1Hgnm-1df6RV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "474435c2-17ec-42c4-f1d8-f88cce5ca929"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: spacy==3.2.4 in /usr/local/lib/python3.7/dist-packages (3.2.4)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (21.3)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (4.64.0)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (3.3.0)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (2.0.6)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (2.11.3)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.8 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (3.0.9)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (57.4.0)\n",
            "Requirement already satisfied: click<8.1.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (7.1.2)\n",
            "Requirement already satisfied: thinc<8.1.0,>=8.0.12 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (8.0.15)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (1.21.6)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (1.0.7)\n",
            "Requirement already satisfied: typer<0.5.0,>=0.3.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (0.4.1)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (2.23.0)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (0.4.1)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (3.0.6)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (1.0.2)\n",
            "Requirement already satisfied: pathy>=0.3.5 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (0.6.1)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.1 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (2.4.3)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (1.8.2)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (0.9.1)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (2.0.7)\n",
            "Requirement already satisfied: typing-extensions<4.0.0.0,>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (3.10.0.2)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from catalogue<2.1.0,>=2.0.6->spacy==3.2.4) (3.8.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->spacy==3.2.4) (3.0.8)\n",
            "Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in /usr/local/lib/python3.7/dist-packages (from pathy>=0.3.5->spacy==3.2.4) (5.2.1)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy==3.2.4) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy==3.2.4) (2021.10.8)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy==3.2.4) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy==3.2.4) (3.0.4)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->spacy==3.2.4) (2.0.1)\n",
            "Collecting ru-core-news-sm==3.2.0\n",
            "  Downloading https://github.com/explosion/spacy-models/releases/download/ru_core_news_sm-3.2.0/ru_core_news_sm-3.2.0-py3-none-any.whl (16.4 MB)\n",
            "\u001b[K     |████████████████████████████████| 16.4 MB 43 kB/s \n",
            "\u001b[?25hRequirement already satisfied: spacy<3.3.0,>=3.2.0 in /usr/local/lib/python3.7/dist-packages (from ru-core-news-sm==3.2.0) (3.2.4)\n",
            "Requirement already satisfied: pymorphy2>=0.9 in /usr/local/lib/python3.7/dist-packages (from ru-core-news-sm==3.2.0) (0.9.1)\n",
            "Requirement already satisfied: docopt>=0.6 in /usr/local/lib/python3.7/dist-packages (from pymorphy2>=0.9->ru-core-news-sm==3.2.0) (0.6.2)\n",
            "Requirement already satisfied: dawg-python>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from pymorphy2>=0.9->ru-core-news-sm==3.2.0) (0.7.2)\n",
            "Requirement already satisfied: pymorphy2-dicts-ru<3.0,>=2.4 in /usr/local/lib/python3.7/dist-packages (from pymorphy2>=0.9->ru-core-news-sm==3.2.0) (2.4.417127.4579844)\n",
            "Requirement already satisfied: click<8.1.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (7.1.2)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (3.0.6)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (1.21.6)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (1.0.7)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (0.9.1)\n",
            "Requirement already satisfied: typer<0.5.0,>=0.3.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (0.4.1)\n",
            "Requirement already satisfied: pathy>=0.3.5 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (0.6.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (2.11.3)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (1.8.2)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (0.4.1)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (4.64.0)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (1.0.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (57.4.0)\n",
            "Requirement already satisfied: thinc<8.1.0,>=8.0.12 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (8.0.15)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (2.0.6)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.8 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (3.0.9)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (2.0.7)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (21.3)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.1 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (2.4.3)\n",
            "Requirement already satisfied: typing-extensions<4.0.0.0,>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (3.10.0.2)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (2.23.0)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (3.3.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from catalogue<2.1.0,>=2.0.6->spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (3.8.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (3.0.8)\n",
            "Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in /usr/local/lib/python3.7/dist-packages (from pathy>=0.3.5->spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (5.2.1)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (2021.10.8)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (2.10)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (2.0.1)\n",
            "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the package via spacy.load('ru_core_news_sm')\n"
          ]
        }
      ],
      "source": [
        "!pip install spacy==3.2.4\n",
        "!python -m spacy download ru_core_news_sm"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Вы можете использовать модель *ru_core_news_lg*, если хотите."
      ],
      "metadata": {
        "id": "56Cq_NVOzKtD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import spacy\n",
        "nlp = spacy.load(\"ru_core_news_sm\")"
      ],
      "metadata": {
        "id": "C-VYPiJ_OUyw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1.1 Токенизация\n",
        "\n",
        "Посчитайте кол-во токенов в тексте \"Каштанка\"\n",
        "1.   Сколько всего токенов в тексте.\n",
        "2.   Сколько токенов в тексте, очищенном от стоп-слов и пунктуации.\n",
        "\n"
      ],
      "metadata": {
        "id": "Ecl7LcGMOSuK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "with open(\"sample_data/Каштанка.txt\", \"r\") as f:\n",
        "    text = f.read()\n",
        "\n",
        "doc = nlp(text)\n",
        "\n",
        "print(\"Всего в тексте {} токенов\".format(len(doc)))\n",
        "\n",
        "clean_doc = []\n",
        "for token in doc:\n",
        "  if not token.is_stop and not token.is_punct:\n",
        "    clean_doc.append(token)\n",
        "\n",
        "print(\"В очищенном тексте {} токенов\".format(len(clean_doc)))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CsRvUyyBOygx",
        "outputId": "70095803-f680-4564-c450-4db0bd43a64b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Всего в тексте 9510 токенов\n",
            "В очищенном тексте 5833 токенов\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1.2 Лемматизация\n",
        "Лемматизируйте данное предложение (выведите все леммы)"
      ],
      "metadata": {
        "id": "pWCTm-b-PTHB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"Она отлично помнила, как она провела день и как в конце концов попала на этот незнакомый тротуар.\"\n",
        "\n",
        "doc = nlp(sentence)\n",
        "\n",
        "for token in doc:\n",
        "  print(token.lemma_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0jncvtouPFFG",
        "outputId": "f2e9b3d4-f4c1-4ade-bdf8-da39c320599f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "она\n",
            "отлично\n",
            "помнить\n",
            ",\n",
            "как\n",
            "она\n",
            "провести\n",
            "день\n",
            "и\n",
            "как\n",
            "в\n",
            "конец\n",
            "конец\n",
            "попасть\n",
            "на\n",
            "этот\n",
            "незнакомый\n",
            "тротуар\n",
            ".\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1.3 Выведите члены предложения \n",
        "Сделать это надо для текста выше (переменная `sentence`)"
      ],
      "metadata": {
        "id": "_wwvlR8hP4mh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for token in doc:\n",
        "  print(token.text,'---- ',token.pos_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_maXKq2WPZg0",
        "outputId": "969cc1d6-040e-48c1-eadd-c4dcdbb74955"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Она ----  PRON\n",
            "отлично ----  ADV\n",
            "помнила ----  VERB\n",
            ", ----  PUNCT\n",
            "как ----  SCONJ\n",
            "она ----  PRON\n",
            "провела ----  VERB\n",
            "день ----  NOUN\n",
            "и ----  CCONJ\n",
            "как ----  SCONJ\n",
            "в ----  ADV\n",
            "конце ----  NOUN\n",
            "концов ----  NOUN\n",
            "попала ----  VERB\n",
            "на ----  ADP\n",
            "этот ----  DET\n",
            "незнакомый ----  ADJ\n",
            "тротуар ----  NOUN\n",
            ". ----  PUNCT\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1.4 Определите схожесть текстов\n",
        "С помощью библиотеки spaCy определите оценку схожести текстов \"Волшебник страны ОЗ\" и \"Волшебник изумрудного города\""
      ],
      "metadata": {
        "id": "f_a3bB0cQNwp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!python -m spacy download ru_core_news_lg"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DHpT2nPrbp2-",
        "outputId": "bc485b64-ffec-428f-c1bb-4e3e31591096"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting ru-core-news-lg==3.2.0\n",
            "  Downloading https://github.com/explosion/spacy-models/releases/download/ru_core_news_lg-3.2.0/ru_core_news_lg-3.2.0-py3-none-any.whl (514.5 MB)\n",
            "\u001b[K     |████████████████████████████████| 514.5 MB 8.6 kB/s \n",
            "\u001b[?25hRequirement already satisfied: pymorphy2>=0.9 in /usr/local/lib/python3.7/dist-packages (from ru-core-news-lg==3.2.0) (0.9.1)\n",
            "Requirement already satisfied: spacy<3.3.0,>=3.2.0 in /usr/local/lib/python3.7/dist-packages (from ru-core-news-lg==3.2.0) (3.2.4)\n",
            "Requirement already satisfied: dawg-python>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from pymorphy2>=0.9->ru-core-news-lg==3.2.0) (0.7.2)\n",
            "Requirement already satisfied: docopt>=0.6 in /usr/local/lib/python3.7/dist-packages (from pymorphy2>=0.9->ru-core-news-lg==3.2.0) (0.6.2)\n",
            "Requirement already satisfied: pymorphy2-dicts-ru<3.0,>=2.4 in /usr/local/lib/python3.7/dist-packages (from pymorphy2>=0.9->ru-core-news-lg==3.2.0) (2.4.417127.4579844)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (2.0.7)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (2.23.0)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (0.9.1)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (4.64.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (21.3)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (1.0.2)\n",
            "Requirement already satisfied: thinc<8.1.0,>=8.0.12 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (8.0.15)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.8 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (3.0.9)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (1.21.6)\n",
            "Requirement already satisfied: click<8.1.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (7.1.2)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (3.0.6)\n",
            "Requirement already satisfied: pathy>=0.3.5 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (0.6.1)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (3.3.0)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (2.0.6)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (0.4.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (2.11.3)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (1.8.2)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.1 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (2.4.3)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (1.0.7)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (57.4.0)\n",
            "Requirement already satisfied: typing-extensions<4.0.0.0,>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (3.10.0.2)\n",
            "Requirement already satisfied: typer<0.5.0,>=0.3.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (0.4.1)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from catalogue<2.1.0,>=2.0.6->spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (3.8.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (3.0.8)\n",
            "Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in /usr/local/lib/python3.7/dist-packages (from pathy>=0.3.5->spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (5.2.1)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (2021.10.8)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (2.10)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (2.0.1)\n",
            "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the package via spacy.load('ru_core_news_lg')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "nlp = spacy.load(\"ru_core_news_lg\")\n",
        "\n",
        "with open(\"sample_data/Волшебник изумрудного города.txt\", \"r\") as f:\n",
        "    text1 = f.read()\n",
        "\n",
        "with open(\"sample_data/Волшебник страны ОЗ.txt\", \"r\") as f:\n",
        "    text2 = f.read()\n",
        "\n",
        "token_1=nlp(text1)\n",
        "token_2=nlp(text2)\n",
        "\n",
        "similarity_score=token_1.similarity(token_2)\n",
        "print('Схожесть \"Волшебника изумрудного города\" и \"Волшебника страны ОЗ\": ' + str(similarity_score))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-1mYrL_8RBGO",
        "outputId": "4eadfe0a-371a-4735-9410-f42b20a6ba91"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Схожесть \"Волшебника изумрудного города\" и \"Волшебника страны ОЗ\": 0.9776418757902706\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "UserWarning в аутпуте выше может возникнуть потому, что вы используете модель _sm, а не _lg. В рамках данного задания можете игнорировать это предупреждение."
      ],
      "metadata": {
        "id": "4PkRA2JqRcV8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1.5 Нахождение паттернов\n",
        "Найдите в тексте \"Каштанка\" сочетания **наречия + глагола**"
      ],
      "metadata": {
        "id": "rqT9ZDJUQ4TO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from spacy.matcher import Matcher \n",
        "matcher = Matcher(nlp.vocab)"
      ],
      "metadata": {
        "id": "aumbI-yzSKfN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "my_pattern=[{\"POS\": \"ADV\"}, {\"POS\": \"VERB\"}]\n",
        "\n",
        "matcher.add(\"VersionFinder\", [my_pattern])\n",
        "\n",
        "doc = nlp(text)\n",
        "\n",
        "desired_matches = matcher(doc)\n",
        "\n",
        "for i in desired_matches:\n",
        "  match = doc[i[1]:i[2]]\n",
        "  print(match.text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0Mmk2OTnRbGH",
        "outputId": "421a7cfa-7db0-440f-b147-3bbcb8f94bda"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "отлично помнила\n",
            "сладко потянулась\n",
            "глубоко вздыхая\n",
            "широко улыбнулся\n",
            "где оставила\n",
            "вперед проходили\n",
            "то спешили\n",
            "так жить\n",
            "совсем облепил\n",
            "уже сидела\n",
            "великолепно пахнет\n",
            "много есть\n",
            "обыкновенно играл\n",
            "сильно дергал\n",
            "тоскливее скулила\n",
            "добродушно понюхали\n",
            "тоже зашипел\n",
            "громко залаяла\n",
            "больно долбанул\n",
            "покорно лег\n",
            "обиженно заскулила\n",
            "Надо жить\n",
            "неподвижно сидел\n",
            "удивленно пятился\n",
            "Немного погодя\n",
            "грубо сколоченного\n",
            "Теперь умри\n",
            "вдруг схватил\n",
            "Теперь представь\n",
            "очень понравился\n",
            "весело захрюкала\n",
            "слегка толкнула\n",
            "сразу поняла\n",
            "лениво потянулся\n",
            "точно делая\n",
            "долго объяснял\n",
            "потом скомандовал\n",
            "тоже свалился\n",
            "брезгливо фыркнул\n",
            "уже водворена\n",
            "уже привыкла\n",
            "брезгливо фыркал\n",
            "тяжело дышал\n",
            "то видела\n",
            "уже свыклась\n",
            "ужасно нравилось\n",
            "так привыкла\n",
            "громко вздохнул\n",
            "теперь пойти\n",
            "Очень может\n",
            "нельзя выходить\n",
            "поскорее уснуть\n",
            "скорее уснешь\n",
            "скорее наступит\n",
            "опять раздался\n",
            "опять хрюкнула\n",
            "очень утомился\n",
            "опять наступили\n",
            "опять стало\n",
            "Опять отворилась\n",
            "так беспокоятся\n",
            "редко покидавший\n",
            "подозрительно заглядывал\n",
            "опять пошел\n",
            "нельзя сделать\n",
            "вниз блестящие\n",
            "уже нет\n",
            "немного погодя\n",
            "еще сказать\n",
            "опять вышел\n",
            "уже сидела\n",
            "беспокойно задвигалась\n",
            "нечаянно высунула\n",
            "плохо освещенную\n",
            "сипло замяукал\n",
            "потом сел\n",
            "густо намазал\n",
            "непременно встретится\n",
            "далеко играла\n",
            "громко крикнул\n",
            "потом вырвалась\n",
            "радостно завыла\n",
            "равнодушно поглядывал\n",
            "глубоко презирал\n",
            "беспокойно задвигалась\n",
            "громко ахнул\n",
            "громко позвали\n",
            "раньше ударил\n",
            "потом вскочила\n",
            "уже идет\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 2. NLTK"
      ],
      "metadata": {
        "id": "bCp8s-rpSVe0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk"
      ],
      "metadata": {
        "id": "KF-OLjjmScRs"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2.1 Частотный словарь\n",
        "Вывести топ-10 часто встречаемых токенов в тексте \"Джек\"."
      ],
      "metadata": {
        "id": "Q_LP31-Vv5H5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from nltk import FreqDist\n",
        "nltk.download('punkt')\n",
        "nltk.download('stopwords')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vOivcCsYSctr",
        "outputId": "8a057fff-3ab8-4a10-fe07-43bf7481bbff"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 94
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def remove_stop_words(tokens: str, stop_words: list) -> list:\n",
        "  clean_tokens = []\n",
        "  for token in tokens:\n",
        "    if token not in stop_words:\n",
        "      clean_tokens.append(token)\n",
        "  return clean_tokens"
      ],
      "metadata": {
        "id": "A-H44PrlgB6U"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "with open(\"sample_data/Джек.txt\", \"r\") as f:\n",
        "    text = f.read()\n",
        "\n",
        "russian_stop_words = nltk.corpus.stopwords.words('russian')\n",
        "\n",
        "tokens = nltk.tokenize.word_tokenize(text, language='russian')\n",
        "\n",
        "clean_tokens = remove_stop_words(tokens, russian_stop_words)"
      ],
      "metadata": {
        "id": "d2Y0X7DbwPCO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "frequency_distribution = FreqDist(clean_tokens) \n",
        "frequency_distribution.most_common(10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zIFKUkwHgOCk",
        "outputId": "c87fd7f2-b1b0-4ea4-85a8-b6eb02f7ebf9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(',', 46),\n",
              " ('Который', 22),\n",
              " ('Которая', 18),\n",
              " ('построил', 9),\n",
              " ('Джек', 9),\n",
              " ('.', 9),\n",
              " ('тёмном', 8),\n",
              " ('чулане', 8),\n",
              " ('хранится', 8),\n",
              " ('В', 8)]"
            ]
          },
          "metadata": {},
          "execution_count": 97
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2.2 Выведите ngrams\n",
        "Выведите 3-грамы из текста \"Джек\""
      ],
      "metadata": {
        "id": "bnnpwbWXwecN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from nltk import ngrams"
      ],
      "metadata": {
        "id": "xkZ4ZvUDxQvH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "n_grams = ngrams(clean_tokens, 3) \n",
        "\n",
        "for i in n_grams:\n",
        "  print(i)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dfoiEuU6waH9",
        "outputId": "592af99a-6967-4870-c4c6-8588b0ee1616"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "('Вот', 'дом', ',')\n",
            "('дом', ',', 'Который')\n",
            "(',', 'Который', 'построил')\n",
            "('Который', 'построил', 'Джек')\n",
            "('построил', 'Джек', '.')\n",
            "('Джек', '.', 'А')\n",
            "('.', 'А', 'это')\n",
            "('А', 'это', 'пшеница')\n",
            "('это', 'пшеница', ',')\n",
            "('пшеница', ',', 'Которая')\n",
            "(',', 'Которая', 'тёмном')\n",
            "('Которая', 'тёмном', 'чулане')\n",
            "('тёмном', 'чулане', 'хранится')\n",
            "('чулане', 'хранится', 'В')\n",
            "('хранится', 'В', 'доме')\n",
            "('В', 'доме', ',')\n",
            "('доме', ',', 'Который')\n",
            "(',', 'Который', 'построил')\n",
            "('Который', 'построил', 'Джек')\n",
            "('построил', 'Джек', '.')\n",
            "('Джек', '.', 'А')\n",
            "('.', 'А', 'это')\n",
            "('А', 'это', 'весёлая')\n",
            "('это', 'весёлая', 'птица-синица')\n",
            "('весёлая', 'птица-синица', ',')\n",
            "('птица-синица', ',', 'Которая')\n",
            "(',', 'Которая', 'часто')\n",
            "('Которая', 'часто', 'ворует')\n",
            "('часто', 'ворует', 'пшеницу')\n",
            "('ворует', 'пшеницу', ',')\n",
            "('пшеницу', ',', 'Которая')\n",
            "(',', 'Которая', 'тёмном')\n",
            "('Которая', 'тёмном', 'чулане')\n",
            "('тёмном', 'чулане', 'хранится')\n",
            "('чулане', 'хранится', 'В')\n",
            "('хранится', 'В', 'доме')\n",
            "('В', 'доме', ',')\n",
            "('доме', ',', 'Который')\n",
            "(',', 'Который', 'построил')\n",
            "('Который', 'построил', 'Джек')\n",
            "('построил', 'Джек', '.')\n",
            "('Джек', '.', 'Вот')\n",
            "('.', 'Вот', 'кот')\n",
            "('Вот', 'кот', ',')\n",
            "('кот', ',', 'Который')\n",
            "(',', 'Который', 'пугает')\n",
            "('Который', 'пугает', 'ловит')\n",
            "('пугает', 'ловит', 'синицу')\n",
            "('ловит', 'синицу', ',')\n",
            "('синицу', ',', 'Которая')\n",
            "(',', 'Которая', 'часто')\n",
            "('Которая', 'часто', 'ворует')\n",
            "('часто', 'ворует', 'пшеницу')\n",
            "('ворует', 'пшеницу', ',')\n",
            "('пшеницу', ',', 'Которая')\n",
            "(',', 'Которая', 'тёмном')\n",
            "('Которая', 'тёмном', 'чулане')\n",
            "('тёмном', 'чулане', 'хранится')\n",
            "('чулане', 'хранится', 'В')\n",
            "('хранится', 'В', 'доме')\n",
            "('В', 'доме', ',')\n",
            "('доме', ',', 'Который')\n",
            "(',', 'Который', 'построил')\n",
            "('Который', 'построил', 'Джек')\n",
            "('построил', 'Джек', '.')\n",
            "('Джек', '.', 'Вот')\n",
            "('.', 'Вот', 'пёс')\n",
            "('Вот', 'пёс', 'хвоста')\n",
            "('пёс', 'хвоста', ',')\n",
            "('хвоста', ',', 'Который')\n",
            "(',', 'Который', 'шиворот')\n",
            "('Который', 'шиворот', 'треплет')\n",
            "('шиворот', 'треплет', 'кота')\n",
            "('треплет', 'кота', ',')\n",
            "('кота', ',', 'Который')\n",
            "(',', 'Который', 'пугает')\n",
            "('Который', 'пугает', 'ловит')\n",
            "('пугает', 'ловит', 'синицу')\n",
            "('ловит', 'синицу', ',')\n",
            "('синицу', ',', 'Которая')\n",
            "(',', 'Которая', 'часто')\n",
            "('Которая', 'часто', 'ворует')\n",
            "('часто', 'ворует', 'пшеницу')\n",
            "('ворует', 'пшеницу', ',')\n",
            "('пшеницу', ',', 'Которая')\n",
            "(',', 'Которая', 'тёмном')\n",
            "('Которая', 'тёмном', 'чулане')\n",
            "('тёмном', 'чулане', 'хранится')\n",
            "('чулане', 'хранится', 'В')\n",
            "('хранится', 'В', 'доме')\n",
            "('В', 'доме', ',')\n",
            "('доме', ',', 'Который')\n",
            "(',', 'Который', 'построил')\n",
            "('Который', 'построил', 'Джек')\n",
            "('построил', 'Джек', '.')\n",
            "('Джек', '.', 'А')\n",
            "('.', 'А', 'это')\n",
            "('А', 'это', 'корова')\n",
            "('это', 'корова', 'безрогая')\n",
            "('корова', 'безрогая', ',')\n",
            "('безрогая', ',', 'Лягнувшая')\n",
            "(',', 'Лягнувшая', 'старого')\n",
            "('Лягнувшая', 'старого', 'пса')\n",
            "('старого', 'пса', 'хвоста')\n",
            "('пса', 'хвоста', ',')\n",
            "('хвоста', ',', 'Который')\n",
            "(',', 'Который', 'шиворот')\n",
            "('Который', 'шиворот', 'треплет')\n",
            "('шиворот', 'треплет', 'кота')\n",
            "('треплет', 'кота', ',')\n",
            "('кота', ',', 'Который')\n",
            "(',', 'Который', 'пугает')\n",
            "('Который', 'пугает', 'ловит')\n",
            "('пугает', 'ловит', 'синицу')\n",
            "('ловит', 'синицу', ',')\n",
            "('синицу', ',', 'Которая')\n",
            "(',', 'Которая', 'часто')\n",
            "('Которая', 'часто', 'ворует')\n",
            "('часто', 'ворует', 'пшеницу')\n",
            "('ворует', 'пшеницу', ',')\n",
            "('пшеницу', ',', 'Которая')\n",
            "(',', 'Которая', 'тёмном')\n",
            "('Которая', 'тёмном', 'чулане')\n",
            "('тёмном', 'чулане', 'хранится')\n",
            "('чулане', 'хранится', 'В')\n",
            "('хранится', 'В', 'доме')\n",
            "('В', 'доме', ',')\n",
            "('доме', ',', 'Который')\n",
            "(',', 'Который', 'построил')\n",
            "('Который', 'построил', 'Джек')\n",
            "('построил', 'Джек', '.')\n",
            "('Джек', '.', 'А')\n",
            "('.', 'А', 'это')\n",
            "('А', 'это', 'старушка')\n",
            "('это', 'старушка', ',')\n",
            "('старушка', ',', 'седая')\n",
            "(',', 'седая', 'строгая')\n",
            "('седая', 'строгая', ',')\n",
            "('строгая', ',', 'Которая')\n",
            "(',', 'Которая', 'доит')\n",
            "('Которая', 'доит', 'корову')\n",
            "('доит', 'корову', 'безрогую')\n",
            "('корову', 'безрогую', ',')\n",
            "('безрогую', ',', 'Лягнувшую')\n",
            "(',', 'Лягнувшую', 'старого')\n",
            "('Лягнувшую', 'старого', 'пса')\n",
            "('старого', 'пса', 'хвоста')\n",
            "('пса', 'хвоста', ',')\n",
            "('хвоста', ',', 'Который')\n",
            "(',', 'Который', 'шиворот')\n",
            "('Который', 'шиворот', 'треплет')\n",
            "('шиворот', 'треплет', 'кота')\n",
            "('треплет', 'кота', ',')\n",
            "('кота', ',', 'Который')\n",
            "(',', 'Который', 'пугает')\n",
            "('Который', 'пугает', 'ловит')\n",
            "('пугает', 'ловит', 'синицу')\n",
            "('ловит', 'синицу', ',')\n",
            "('синицу', ',', 'Которая')\n",
            "(',', 'Которая', 'часто')\n",
            "('Которая', 'часто', 'ворует')\n",
            "('часто', 'ворует', 'пшеницу')\n",
            "('ворует', 'пшеницу', ',')\n",
            "('пшеницу', ',', 'Которая')\n",
            "(',', 'Которая', 'тёмном')\n",
            "('Которая', 'тёмном', 'чулане')\n",
            "('тёмном', 'чулане', 'хранится')\n",
            "('чулане', 'хранится', 'В')\n",
            "('хранится', 'В', 'доме')\n",
            "('В', 'доме', ',')\n",
            "('доме', ',', 'Который')\n",
            "(',', 'Который', 'построил')\n",
            "('Который', 'построил', 'Джек')\n",
            "('построил', 'Джек', '.')\n",
            "('Джек', '.', 'А')\n",
            "('.', 'А', 'это')\n",
            "('А', 'это', 'ленивый')\n",
            "('это', 'ленивый', 'толстый')\n",
            "('ленивый', 'толстый', 'пастух')\n",
            "('толстый', 'пастух', ',')\n",
            "('пастух', ',', 'Который')\n",
            "(',', 'Который', 'бранится')\n",
            "('Который', 'бранится', 'коровницей')\n",
            "('бранится', 'коровницей', 'строгою')\n",
            "('коровницей', 'строгою', ',')\n",
            "('строгою', ',', 'Которая')\n",
            "(',', 'Которая', 'доит')\n",
            "('Которая', 'доит', 'корову')\n",
            "('доит', 'корову', 'безрогую')\n",
            "('корову', 'безрогую', ',')\n",
            "('безрогую', ',', 'Лягнувшую')\n",
            "(',', 'Лягнувшую', 'старого')\n",
            "('Лягнувшую', 'старого', 'пса')\n",
            "('старого', 'пса', 'хвоста')\n",
            "('пса', 'хвоста', ',')\n",
            "('хвоста', ',', 'Который')\n",
            "(',', 'Который', 'шиворот')\n",
            "('Который', 'шиворот', 'треплет')\n",
            "('шиворот', 'треплет', 'кота')\n",
            "('треплет', 'кота', ',')\n",
            "('кота', ',', 'Который')\n",
            "(',', 'Который', 'пугает')\n",
            "('Который', 'пугает', 'ловит')\n",
            "('пугает', 'ловит', 'синицу')\n",
            "('ловит', 'синицу', ',')\n",
            "('синицу', ',', 'Которая')\n",
            "(',', 'Которая', 'часто')\n",
            "('Которая', 'часто', 'ворует')\n",
            "('часто', 'ворует', 'пшеницу')\n",
            "('ворует', 'пшеницу', ',')\n",
            "('пшеницу', ',', 'Которая')\n",
            "(',', 'Которая', 'тёмном')\n",
            "('Которая', 'тёмном', 'чулане')\n",
            "('тёмном', 'чулане', 'хранится')\n",
            "('чулане', 'хранится', 'В')\n",
            "('хранится', 'В', 'доме')\n",
            "('В', 'доме', ',')\n",
            "('доме', ',', 'Который')\n",
            "(',', 'Который', 'построил')\n",
            "('Который', 'построил', 'Джек')\n",
            "('построил', 'Джек', '.')\n",
            "('Джек', '.', 'Вот')\n",
            "('.', 'Вот', 'петуха')\n",
            "('Вот', 'петуха', ',')\n",
            "('петуха', ',', 'Которые')\n",
            "(',', 'Которые', 'будят')\n",
            "('Которые', 'будят', 'пастуха')\n",
            "('будят', 'пастуха', ',')\n",
            "('пастуха', ',', 'Который')\n",
            "(',', 'Который', 'бранится')\n",
            "('Который', 'бранится', 'коровницей')\n",
            "('бранится', 'коровницей', 'строгою')\n",
            "('коровницей', 'строгою', ',')\n",
            "('строгою', ',', 'Которая')\n",
            "(',', 'Которая', 'доит')\n",
            "('Которая', 'доит', 'корову')\n",
            "('доит', 'корову', 'безрогую')\n",
            "('корову', 'безрогую', ',')\n",
            "('безрогую', ',', 'Лягнувшую')\n",
            "(',', 'Лягнувшую', 'старого')\n",
            "('Лягнувшую', 'старого', 'пса')\n",
            "('старого', 'пса', 'хвоста')\n",
            "('пса', 'хвоста', ',')\n",
            "('хвоста', ',', 'Который')\n",
            "(',', 'Который', 'шиворот')\n",
            "('Который', 'шиворот', 'треплет')\n",
            "('шиворот', 'треплет', 'кота')\n",
            "('треплет', 'кота', ',')\n",
            "('кота', ',', 'Который')\n",
            "(',', 'Который', 'пугает')\n",
            "('Который', 'пугает', 'ловит')\n",
            "('пугает', 'ловит', 'синицу')\n",
            "('ловит', 'синицу', ',')\n",
            "('синицу', ',', 'Которая')\n",
            "(',', 'Которая', 'часто')\n",
            "('Которая', 'часто', 'ворует')\n",
            "('часто', 'ворует', 'пшеницу')\n",
            "('ворует', 'пшеницу', ',')\n",
            "('пшеницу', ',', 'Которая')\n",
            "(',', 'Которая', 'тёмном')\n",
            "('Которая', 'тёмном', 'чулане')\n",
            "('тёмном', 'чулане', 'хранится')\n",
            "('чулане', 'хранится', 'В')\n",
            "('хранится', 'В', 'доме')\n",
            "('В', 'доме', ',')\n",
            "('доме', ',', 'Который')\n",
            "(',', 'Который', 'построил')\n",
            "('Который', 'построил', 'Джек')\n",
            "('построил', 'Джек', '.')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2.3 Тональность текста"
      ],
      "metadata": {
        "id": "nr2GbmMBxU0N"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Выведите оценку тональности текста из таблицы \"Harry Potter 1.csv\""
      ],
      "metadata": {
        "id": "hhGkI5rmzjGi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "from nltk.sentiment.vader import SentimentIntensityAnalyzer\n",
        "nltk.download('vader_lexicon')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EyskB8SpxZiu",
        "outputId": "d40dd086-7f47-4baa-9f94-6ed8d9a19f31"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package vader_lexicon to /root/nltk_data...\n",
            "[nltk_data]   Package vader_lexicon is already up-to-date!\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 100
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df = pd.read_csv('sample_data/Harry Potter 1.csv', encoding= 'unicode_escape', sep='\\t')\n",
        "df_lines = df.groupby('Character').count()\n",
        "df_lines = df_lines.sort_values(by=['Sentence'], ascending=False)[:10]\n",
        "top_character_names = df_lines.index.values\n",
        "\n",
        "df_character_sentiment = df[df['Character'].isin(top_character_names)]\n",
        "df_character_sentiment = df_character_sentiment[['Character', 'Sentence']]\n",
        "\n",
        "sid = SentimentIntensityAnalyzer()\n",
        "df_character_sentiment.reset_index(inplace=True, drop=True)\n",
        "df_character_sentiment[['neg', 'neu', 'pos', 'compound']] = df_character_sentiment['Sentence'].apply(sid.polarity_scores).apply(pd.Series)\n",
        "df_character_sentiment"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "OhA28DWLxlEc",
        "outputId": "9954cfe2-475b-4230-a805-752ee39dbd43"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "      Character                                           Sentence    neg  \\\n",
              "0    Dumbledore  I should've known that you would be here, Prof...  0.000   \n",
              "1    McGonagall                Good evening, Professor Dumbledore.  0.000   \n",
              "2    McGonagall                        Are the rumors true, Albus?  0.000   \n",
              "3    Dumbledore                          I'm afraid so, professor.  0.000   \n",
              "4    Dumbledore                              The good and the bad.  0.372   \n",
              "..          ...                                                ...    ...   \n",
              "903      Hagrid  Oh, listen, Harry, if that dolt of a cousin of...  0.150   \n",
              "904       Harry  But Hagrid, we're not allowed to do magic away...  0.000   \n",
              "905      Hagrid  I do. But your cousin don't, do he? Eh? Off yo...  0.000   \n",
              "906    Hermione        Feels strange to be going home, doesn't it?  0.205   \n",
              "907       Harry                    I'm not going home. Not really.  0.000   \n",
              "\n",
              "       neu    pos  compound  \n",
              "0    1.000  0.000    0.0000  \n",
              "1    0.508  0.492    0.4404  \n",
              "2    0.588  0.412    0.4215  \n",
              "3    1.000  0.000    0.0000  \n",
              "4    0.319  0.309   -0.1531  \n",
              "..     ...    ...       ...  \n",
              "903  0.777  0.073   -0.4588  \n",
              "904  1.000  0.000    0.0000  \n",
              "905  1.000  0.000    0.0000  \n",
              "906  0.795  0.000   -0.2023  \n",
              "907  1.000  0.000    0.0000  \n",
              "\n",
              "[908 rows x 6 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-65d1d71f-ca58-471e-8dbc-3c0e65bbefdb\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Character</th>\n",
              "      <th>Sentence</th>\n",
              "      <th>neg</th>\n",
              "      <th>neu</th>\n",
              "      <th>pos</th>\n",
              "      <th>compound</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Dumbledore</td>\n",
              "      <td>I should've known that you would be here, Prof...</td>\n",
              "      <td>0.000</td>\n",
              "      <td>1.000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.0000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>McGonagall</td>\n",
              "      <td>Good evening, Professor Dumbledore.</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.508</td>\n",
              "      <td>0.492</td>\n",
              "      <td>0.4404</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>McGonagall</td>\n",
              "      <td>Are the rumors true, Albus?</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.588</td>\n",
              "      <td>0.412</td>\n",
              "      <td>0.4215</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Dumbledore</td>\n",
              "      <td>I'm afraid so, professor.</td>\n",
              "      <td>0.000</td>\n",
              "      <td>1.000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.0000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Dumbledore</td>\n",
              "      <td>The good and the bad.</td>\n",
              "      <td>0.372</td>\n",
              "      <td>0.319</td>\n",
              "      <td>0.309</td>\n",
              "      <td>-0.1531</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>903</th>\n",
              "      <td>Hagrid</td>\n",
              "      <td>Oh, listen, Harry, if that dolt of a cousin of...</td>\n",
              "      <td>0.150</td>\n",
              "      <td>0.777</td>\n",
              "      <td>0.073</td>\n",
              "      <td>-0.4588</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>904</th>\n",
              "      <td>Harry</td>\n",
              "      <td>But Hagrid, we're not allowed to do magic away...</td>\n",
              "      <td>0.000</td>\n",
              "      <td>1.000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.0000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>905</th>\n",
              "      <td>Hagrid</td>\n",
              "      <td>I do. But your cousin don't, do he? Eh? Off yo...</td>\n",
              "      <td>0.000</td>\n",
              "      <td>1.000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.0000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>906</th>\n",
              "      <td>Hermione</td>\n",
              "      <td>Feels strange to be going home, doesn't it?</td>\n",
              "      <td>0.205</td>\n",
              "      <td>0.795</td>\n",
              "      <td>0.000</td>\n",
              "      <td>-0.2023</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>907</th>\n",
              "      <td>Harry</td>\n",
              "      <td>I'm not going home. Not really.</td>\n",
              "      <td>0.000</td>\n",
              "      <td>1.000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.0000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>908 rows × 6 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-65d1d71f-ca58-471e-8dbc-3c0e65bbefdb')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-65d1d71f-ca58-471e-8dbc-3c0e65bbefdb button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-65d1d71f-ca58-471e-8dbc-3c0e65bbefdb');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 111
        }
      ]
    }
  ]
}